Skip to main content
SpringerLink
Log in

A Knowledge Infusion Based Multitasking System for Sarcasm Detection in Meme

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13980))

Included in the following conference series:

Abstract

In this paper, we hypothesize that sarcasm detection is closely associated with the emotion present in memes. Thereafter, we propose a deep multitask model to perform these two tasks in parallel, where sarcasm detection is treated as the primary task, and emotion recognition is considered an auxiliary task. We create a large-scale dataset consisting of 7416 memes in Hindi, one of the widely spoken languages. We collect the memes from various domains, such as politics, religious, racist, and sexist, and manually annotate each instance with three sarcasm categories, i.e., i) Not Sarcastic, ii) Mildly Sarcastic or iii) Highly Sarcastic and 13 fine-grained emotion classes. Furthermore, we propose a novel Knowledge Infusion (KI) based module which captures sentiment-aware representation from a pre-trained model using the Memotion dataset. Detailed empirical evaluation shows that the multitasking model performs better than the single-task model. We also show that using this KI module on top of our model can boost the performance of sarcasm detection in both single-task and multi-task settings even further. Code and dataset are available at this link: https://www.iitp.ac.in/ ai-nlp-ml/resources.html#Sarcastic-Meme-Detection.

D. Bandyopadhyay and G. Kumari—Equal Contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    github.com/tesseract-ocr/tesseract.

  2. 2.

    https://github.com/FreddeFrallan/Multilingual-CLIP.

  3. 3.

    https://competitions.codalab.org/competitions/35688.

  4. 4.

    Each meme in Memotion. dataset is annotated with both sarcasm and sentiment classes.

References

  1. Bayerl, P.S., Paul, K.I.: What determines inter-coder agreement in manual annotations? A meta-analytic investigation. Comput. Linguist. 37(4), 699–725 (2011). https://doi.org/10.1162/COLI_a_00074, https://aclanthology.org/J11-4004

  2. Bernadt, M., Emmanuel, J.: Diagnostic agreement in psychiatry. Br. J. Psychiatry: J. Mental Sci. 163, 549–50 (1993). https://doi.org/10.1192/S0007125000034012

  3. Boland, K., Wira-Alam, A., Messerschmidt, R.: Creating an annotated corpus for sentiment analysis of German product reviews. GESIS-Technical reports, 2013/05 (2013)

    Google Scholar 

  4. Bouazizi, O.: Tomoaki: a pattern-based approach for sarcasm detection on twitter. IEEE Access 4, 5477–5488 (2016). https://doi.org/10.1109/ACCESS.2016.2594194

    Article  Google Scholar 

  5. Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann, R., Mihalcea, R., Poria, S.: Towards multimodal sarcasm detection (an _obviously_ perfect paper). CoRR abs/1906.01815 (2019). http://arxiv.org/abs/1906.01815

  6. Chauhan, D.S., Dhanush, S.R., Ekbal, A., Bhattacharyya, P.: Sentiment and emotion help sarcasm? A multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4351–4360. Association for Computational Linguistics, Online, July 2020. https://doi.org/10.18653/v1/2020.acl-main.401, https://aclanthology.org/2020.acl-main.401

  7. Davidson, T., Bhattacharya, D., Weber, I.: Racial bias in hate speech and abusive language detection datasets. In: Proceedings of the Third Workshop on Abusive Language Online, pp. 25–35. Association for Computational Linguistics, Florence, August 2019. https://doi.org/10.18653/v1/W19-3504, https://aclanthology.org/W19-3504

  8. Dong, X., Li, C., Choi, J.D.: Transformer-based context-aware sarcasm detection in conversation threads from social media. CoRR abs/2005.11424 (2020). https://arxiv.org/abs/2005.11424

  9. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale (2020). https://doi.org/10.48550/ARXIV.2010.11929, https://arxiv.org/abs/2010.11929

  10. Dror, R., Baumer, G., Shlomov, S., Reichart, R.: The Hitchhiker’s guide to testing statistical significance in natural language processing. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1383–1392. Association for Computational Linguistics, Melbourne, Australia, July 2018. https://doi.org/10.18653/v1/P18-1128, https://aclanthology.org/P18-1128

  11. Ekman, P., Cordaro, D.T.: What is meant by calling emotions basic. Emot. Rev. 3, 364–370 (2011)

    Article  Google Scholar 

  12. Ghosal, D., Akhtar, M.S., Chauhan, D., Poria, S., Ekbal, A., Bhattacharyya, P.: Contextual inter-modal attention for multi-modal sentiment analysis. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3454–3466. Association for Computational Linguistics, Brussels, October–November 2018. https://doi.org/10.18653/v1/D18-1382, https://aclanthology.org/D18-1382

  13. Ghosh, D., Fabbri, A.R., Muresan, S.: The role of conversation context for sarcasm detection in online interactions. CoRR abs/1707.06226 (2017). http://arxiv.org/abs/1707.06226

  14. Hasan, M.K., et al.: UR-FUNNY: a multimodal language dataset for understanding humor. CoRR abs/1904.06618 (2019). http://arxiv.org/abs/1904.06618

  15. He, S., Zheng, X., Wang, J., Chang, Z., Luo, Y., Zeng, D.: Meme extraction and tracing in crisis events. In: 2016 IEEE Conference on Intelligence and Security Informatics (ISI), pp. 61–66 (2016). https://doi.org/10.1109/ISI.2016.7745444

  16. Joshi, A., Bhattacharyya, P., Carman, M.J.: Automatic sarcasm detection: a survey. ACM Comput. Surv. 50(5), 73:1–73:22 (2017). https://doi.org/10.1145/3124420

  17. Joshi, A., Tripathi, V., Bhattacharyya, P., Carman, M.J.: Harnessing sequence labeling for sarcasm detection in dialogue from TV series ‘Friends’. In: Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 146–155. Association for Computational Linguistics, Berlin, August 2016. https://doi.org/10.18653/v1/K16-1015, https://aclanthology.org/K16-1015

  18. Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., Testuggine, D.: The hateful memes challenge: detecting hate speech in multimodal memes. CoRR abs/2005.04790 (2020). https://arxiv.org/abs/2005.04790

  19. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2015)

    Google Scholar 

  20. Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotic: emotions in context dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2309–2317 (2017). https://doi.org/10.1109/CVPRW.2017.285

  21. krippendorff, k.: Computing Krippendorff’s alpha-reliability, January 2011

    Google Scholar 

  22. Kumar, A., Joshi, A.: Striking a balance: alleviating inconsistency in pre-trained models for symmetric classification tasks. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 1887–1895. Association for Computational Linguistics, Dublin, May 2022. https://doi.org/10.18653/v1/2022.findings-acl.148, https://aclanthology.org/2022.findings-acl.148

  23. Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: VisualBERT: a simple and performant baseline for vision and language (2019). https://doi.org/10.48550/ARXIV.1908.03557, https://arxiv.org/abs/1908.03557

  24. Liu, L., Priestley, J.L., Zhou, Y., Ray, H.E., Han, M.: A2text-net: a novel deep neural network for sarcasm detection. In: 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI), pp. 118–126 (2019). https://doi.org/10.1109/CogMI48466.2019.00025

  25. Liu, Y., et al.: Multilingual denoising pre-training for neural machine translation (2020). https://doi.org/10.48550/ARXIV.2001.08210, https://arxiv.org/abs/2001.08210

  26. Majumder, N., Poria, S., Peng, H., Chhaya, N., Cambria, E., Gelbukh, A.: Sentiment and sarcasm classification with multitask learning. IEEE Intell. Syst. 34, 38–43 (2019). https://doi.org/10.1109/MIS.2019.2904691

  27. McNemar, Q.: Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157 (1947). https://doi.org/10.1007/bf02295996

  28. Öhman, E.: Emotion annotation: rethinking emotion categorization. In: DHN Post-Proceedings (2020)

    Google Scholar 

  29. Pires, T., Schlinger, E., Garrette, D.: How multilingual is multilingual BERT? (2019). https://doi.org/10.48550/ARXIV.1906.01502, https://arxiv.org/abs/1906.01502

  30. Poria, S., Cambria, E., Hazarika, D., Vij, P.: A deeper look into sarcastic tweets using deep convolutional neural networks. CoRR abs/1610.08815 (2016). http://arxiv.org/abs/1610.08815

  31. Radford, A., et al.: Learning transferable visual models from natural language supervision. CoRR abs/2103.00020 (2021). https://arxiv.org/abs/2103.00020

  32. Rhanoui, M., Mikram, M., Yousfi, S., Barzali, S.: A CNN-BILSTM model for document-level sentiment analysis. Mach. Learn. Knowl. Extract. 1(3), 832–847 (2019). https://doi.org/10.3390/make1030048, https://www.mdpi.com/2504-4990/1/3/48

  33. Ribeiro, M.T., Singh, S., Guestrin, C.: “why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016, pp. 1135–1144 (2016)

    Google Scholar 

  34. Rockwell, P., Theriot, E.M.: Culture, gender, and gender mix in encoders of sarcasm: a self-assessment analysis. Commun. Res. Rep. 18(1), 44–52 (2001). https://doi.org/10.1080/08824090109384781, https://doi.org/10.1080/08824090109384781

  35. Sharma, C., et al.: SemEval-2020 task 8: memotion analysis - the visuo-lingual metaphor! CoRR abs/2008.03781 (2020). https://arxiv.org/abs/2008.03781

  36. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2015)

    Google Scholar 

  37. Suryawanshi, S., Chakravarthi, B.R., Arcan, M., Buitelaar, P.: Multimodal meme dataset (MultiOFF) for identifying offensive content in image and text. In: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pp. 32–41. European Language Resources Association (ELRA), Marseille, France, May 2020. https://aclanthology.org/2020.trac-1.6

  38. Tan, H., Bansal, M.: LXMERT: learning cross-modality encoder representations from transformers (2019). https://doi.org/10.48550/ARXIV.1908.07490, https://arxiv.org/abs/1908.07490

  39. Tsur, O., Rappoport, A.: RevRank: a fully unsupervised algorithm for selecting the most helpful book reviews. In: ICWSM (2009)

    Google Scholar 

  40. Venkatesan, R., Er, M.J.: Multi-label classification method based on extreme learning machines. In: 2014 13th International Conference on Control Automation Robotics Vision (ICARCV), pp. 619–624 (2014). https://doi.org/10.1109/ICARCV.2014.7064375

  41. Wilson, P.A., Lewandowska, B.: The Nature of Emotions. In: Cambridge University Press (2012)

    Google Scholar 

  42. Zhu, H., Mak, D., Gioannini, J., Xia, F.: NLPStatTest: a toolkit for comparing NLP system performance. In: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 40–46. Association for Computational Linguistics, Suzhou, China, December 2020. https://aclanthology.org/2020.aacl-demo.7

Download references

Acknowledgement

The research reported in this paper is an outcome of the project “HELIOS-Hate, Hyperpartisan, and Hyperpluralism Elicitation and Observer System”, sponsored by Wipro.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology Patna, Bihta, India

    Dibyanayan Bandyopadhyay, Gitanjali Kumari & Asif Ekbal

  2. Wipro AI Labs, Bengaluru, India

    Santanu Pal, Arindam Chatterjee & Vinutha BN

Authors
  1. Dibyanayan Bandyopadhyay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gitanjali Kumari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Asif Ekbal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Santanu Pal
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Arindam Chatterjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Vinutha BN
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dibyanayan Bandyopadhyay .

Editor information

Editors and Affiliations

  1. University of Amsterdam, Amsterdam, The Netherlands

    Jaap Kamps

  2. Université Grenoble-Alpes, Saint-Martin-d’Hères, France

    Lorraine Goeuriot

  3. Università della Svizzera Italiana, Lugano, Switzerland

    Fabio Crestani

  4. University of Copenhagen, Copenhagen, Denmark

    Maria Maistro

  5. University of Tsukuba, Ibaraki, Japan

    Hideo Joho

  6. Dublin City University, Dublin, Ireland

    Brian Davis

  7. Dublin City University, Dublin, Ireland

    Cathal Gurrin

  8. Universität Regensburg, Regensburg, Germany

    Udo Kruschwitz

  9. Dublin City University, Dublin, Ireland

    Annalina Caputo

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bandyopadhyay, D., Kumari, G., Ekbal, A., Pal, S., Chatterjee, A., BN, V. (2023). A Knowledge Infusion Based Multitasking System for Sarcasm Detection in Meme. In: Kamps, J., et al. Advances in Information Retrieval. ECIR 2023. Lecture Notes in Computer Science, vol 13980. Springer, Cham. https://doi.org/10.1007/978-3-031-28244-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-28244-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-28243-0

  • Online ISBN: 978-3-031-28244-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation