Skip to main content
SpringerLink
Log in

Sh-DistilBERT: New Transfer Learning Model for Arabic Sentiment Analysis and Aspect Category Detection

  • Conference paper
  • First Online:
Advances in Computational Collective Intelligence (ICCCI 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1864))

Included in the following conference series:

  • 413 Accesses

Abstract

Arabic sentiment analysis is the process of computationally identifying and categorizing opinions expressed in a piece of text, particularly to assess whether the writer’s attitude toward a given topic, product, etc. is positive, negative, or neutral. In sentiment analysis, aspect category detection (ACD) attempts to identify the aspect categories mentioned in a sentence. Our study investigates the effects of transfer learning across several Arabic NLP tasks. We proposed a new shared DistilBERT model, which is a fine-tuned version of the basic DistilBERT. Our results demonstrate the outperforming of the proposed approach for the two tasks presented in the study, with a small variation. We also showed the limited effects of transfer learning on the performance of the proposed approach, particularly for highly dialectic comments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Mozafari, M., Farahbakhsh, R., Crespi, N.: A BERT-based transfer learning approach for hate speech detection in online social media. In: Cherifi, H., Gaito, S., Mendes, J.F., Moro, E., Rocha, L.M. (eds.) COMPLEX NETWORKS 2019. SCI, vol. 881, pp. 928–940. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-36687-2_77

    Chapter  Google Scholar 

  2. Bensoltane, R., Zaki, T.: Towards Arabic aspect-based sentiment analysis: a transfer learning-based approach. Soc. Netw. Anal. Min. 12, 1–16 (2022)

    Article  Google Scholar 

  3. Chouikhi, H., Alsuhaibani, M.: Deep transformer language models for Arabic text summarization: a comparison study. Appl. Sci. 12(23), 11944 (2022)

    Article  Google Scholar 

  4. Chouikhi, H., Alsuhaibani, M., Jarray, F.: BERT-based joint model for aspect term extraction and aspect polarity detection in Arabic text. Electronics 12(3), 515 (2023)

    Article  Google Scholar 

  5. Al-Smadi, M., Qawasmeh, O., Talafha, B., Quwaider, M.: Human annotated Arabic dataset of book reviews for aspect based sentiment analysis. In: 2015 3rd International Conference on Future Internet of Things and Cloud, pp. 726–730. IEEE, August 2015

    Google Scholar 

  6. Pontiki, M., et al.: Semeval-2016 task 5: aspect based sentiment analysis. In: ProWorkshop on Semantic Evaluation (SemEval-2016), pp. 19–30. Association for Computational Linguistics (2016)

    Google Scholar 

  7. Al-Dabet, S., Tedmori, S., Mohammad, A.S.: Enhancing Arabic aspect-based sentiment analysis using deep learning models. Comput. Speech Lang. 69, 101224 (2021)

    Article  Google Scholar 

  8. Gonnet, P., Deselaers, T.: Indylstms: independently recurrent LSTMs. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3352–3356. IEEE, May 2020

    Google Scholar 

  9. Obaidat, I., Mohawesh, R., Al-Ayyoub, M., Mohammad, A.S., Jararweh, Y. Enhancing the determination of aspect categories and their polarities in Arabic reviews using lexicon-based approaches. In: 2015 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT), pp. 1–6. IEEE, November 2015

    Google Scholar 

  10. Tamchyna, A., Veselovská, K.: Ufal at semeval-2016 task 5: recurrent neural networks for sentence classification. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SEMEVAL-2016), pp. 367–371, June 2016

    Google Scholar 

  11. Bensoltane, R., Zaki, T.: Comparing word embedding models for Arabic aspect category detection using a deep learning-based approach. In: E3S Web of Conferences, vol. 297, p. 01072. EDP Sciences (2021)

    Google Scholar 

  12. Heikal, M., Torki, M., El-Makky, N.: Sentiment analysis of Arabic tweets using deep learning. Procedia Comput. Sci. 142, 114–122 (2018)

    Article  Google Scholar 

  13. Nabil, M., Aly, M., Atiya, A.: Astd: Arabic sentiment tweets dataset. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2515–2519, September 2015

    Google Scholar 

  14. Alayba, A.M., Palade, V., England, M., Iqbal, R.: Arabic language sentiment analysis on health services. In: 2017 1st International Workshop on Arabic Script Analysis and Recognition (ASAR), pp. 114–118. IEEE, April 2017

    Google Scholar 

  15. Alayba, A.M., Palade, V., England, M., Iqbal, R.: A combined CNN and LSTM model for Arabic sentiment analysis. In: Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds.) CD-MAKE 2018. LNCS, vol. 11015, pp. 179–191. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99740-7_12

    Chapter  Google Scholar 

  16. Abdul-Mageed, M., Diab, M., Korayem, M.: Subjectivity and sentiment analysis of modern standard Arabic. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 587–591, June 2011

    Google Scholar 

  17. Abdul-Mageed, M., Diab, M., Kübler, S.: SAMAR: subjectivity and sentiment analysis for Arabic social media. Comput. Speech Lang. 28, 20–37 (2014)

    Article  Google Scholar 

  18. El-Beltagy, S.R., Khalil, T., Halaby, A., Hammad, M.: Combining lexical features and a supervised learning approach for Arabic sentiment analysis. In: Gelbukh, A. (ed.) CICLing 2016. LNCS, vol. 9624, pp. 307–319. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75487-1_24

    Chapter  Google Scholar 

  19. Shoukry, A., Rafea, A.: A hybrid approach for sentiment classification of Egyptian dialect tweets. In: 2015 First International Conference on Arabic Computational Linguistics (ACLing), pp. 78–85. IEEE, April 2015

    Google Scholar 

  20. Al Sallab, A., Hajj, H., Badaro, G., Baly, R., El-Hajj, W., Shaban, K.: Deep learning models for sentiment analysis in Arabic. In: Proceedings of the Second Workshop on Arabic Natural Language Processing, pp. 9–17, July 2015

    Google Scholar 

  21. Baly, R., et al.: A characterization study of Arabic twitter data with a benchmarking for state-of-the-art opinion mining models. In: Proceedings of the third Arabic Natural Language Processing Workshop, pp. 110–118, April 2017

    Google Scholar 

  22. Baly, R., Hajj, H., Habash, N., Shaban, K.B., El-Hajj, W.: A sentiment treebank and morphologically enriched recursive deep models for effective sentiment analysis in Arabic. ACM Trans. Asian Low-Resour. Lang. Inf. Process. (TALLIP) 16(4), 1–21 (2017)

    Article  Google Scholar 

  23. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  24. Rogers, A., Kovaleva, O., Rumshisky, A.: A primer in BERTology: what we know about how BERT works. Trans. Assoc. Comput. Linguist. 8, 842–866 (2021)

    Article  Google Scholar 

  25. Zaib, M., Sheng, Q.Z., Emma Zhang, W.: A short survey of pre-trained language models for conversational AI-a new age in NLP. In: Proceedings of the Australasian Computer Science Week Multiconference, pp. 1–4, February 2020

    Google Scholar 

  26. Alshalan, R., Al-Khalifa, H.: A deep learning approach for automatic hate speech detection in the Saudi twitter sphere. Appl. Sci. 10(23), 8614 (2020)

    Article  Google Scholar 

  27. Antoun, W., Baly, F., Hajj, H.: Arabert: transformer-based model for Arabic language understanding. arXiv preprint arXiv:2003.00104 (2020)

  28. Safaya, A., Abdullatif, M., Yuret, D.: Kuisail at semeval-2020 task 12: bert-cnn for offensive speech identification in social media. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 2054–2059, December 2020

    Google Scholar 

  29. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., Habash, N.: The interplay of variant, size, and task type in Arabic pre-trained language models. arXiv preprint arXiv:2103.06678 (2021)

  30. Abdul-Mageed, M., Elmadany, A., Nagoudi, E.M.B.: ARBERT & MARBERT: deep bidirectional transformers for Arabic. arXiv preprint arXiv:2101.01785 (2020)

  31. Chouikhi, H., Chniter, H., Jarray, F.: Arabic sentiment analysis using BERT model. In: Wojtkiewicz, K., Treur, J., Pimenidis, E., Maleszka, M. (eds.) ICCCI 2021. CCIS, vol. 1463, pp. 621–632. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88113-9_50

    Chapter  Google Scholar 

  32. Chouikhi, H., Chniter, H., Jarray, F.: Stacking BERT based models for Arabic sentiment analysis. In: Proceedings of the 13th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 2: KEOD, pp. 144–150 (2021). https://doi.org/10.5220/0010648400003064, ISBN 978-989-758-533-3, ISSN 2184-3228

  33. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. (2019)

  34. Mohammad, A.S., Qwasmeh, O., Talafha, B., Al-Ayyoub, M., Jararweh, Y., Benkhelifa, E.: An enhanced framework for aspect-based sentiment analysis of hotels’ reviews: Arabic reviews case study. In: 2016 11th International Conference for Internet Technology and Secured Transactions (ICITST), pp. 98–103. IEEE, December 2016

    Google Scholar 

  35. Al-Smadi, M., Talafha, B., Al-Ayyoub, M., Jararweh, Y.: Using long short-term memory deep neural networks for aspect-based sentiment analysis of Arabic reviews. Int. J. Mach. Learn. Cybern. 10, 2163–2175 (2019)

    Article  Google Scholar 

  36. Elnagar, A., Khalifa, Y.S., Einea, A.: Hotel Arabic-reviews dataset construction for sentiment analysis applications. In: Shaalan, K., Hassanien, A.E., Tolba, F. (eds.) Intelligent Natural Language Processing: Trends and Applications. SCI, vol. 740, pp. 35–52. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-67056-0_3

    Chapter  Google Scholar 

  37. Aly, M., Atiya, A.: LABR: A Large Scale Arabic Book Reviews Dataset. Meetings of the Association for Computational Linguistics (ACL) At: Sofia, Bulgaria (2013)

    Google Scholar 

  38. Alomari, K.M., ElSherif, H.M., Shaalan, K.: Arabic tweets sentimental analysis using machine learning. In: Benferhat, S., Tabia, K., Ali, M. (eds.) IEA/AIE 2017. LNCS (LNAI), vol. 10350, pp. 602–610. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60042-0_66

    Chapter  Google Scholar 

  39. Saidi, R., Jarray, F.: Sentence transformers and DistilBERT for Arabic word sense induction. In: Proceedings of the 15th International Conference on Agents and Artificial Intelligence, vol. 3, pp 1020–1027 (2023). ISBN 978-989-758-623-1. ISSN 2184-433X

    Google Scholar 

  40. Saad, M.K., Ashour, W.: Osac: open source Arabic corpora. In: 6th ArchEng International Symposiums, EEECS, vol. 10, November 2010

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LIMTIC Laboratory, UTM University, Tunis, Tunisia

    Hasna Chouikhi

  2. Higher Institute of Computer Science of Medenine, Medenine, Tunisia

    Fethi Jarray

Authors
  1. Hasna Chouikhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fethi Jarray
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hasna Chouikhi .

Editor information

Editors and Affiliations

  1. Wrocław University of Science and Technology, Wrocław, Poland

    Ngoc Thanh Nguyen

  2. Eötvös Loránd University, Budapest, Hungary

    János Botzheim

  3. Eötvös Loránd University, Budapest, Hungary

    László Gulyás

  4. Universidad Complutense de Madrid, Madrid, Spain

    Manuel Nunez

  5. Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

    Jan Treur

  6. University of Münster, Münster, Germany

    Gottfried Vossen

  7. Wrocław University of Science and Technology, Wrocław, Poland

    Adrianna Kozierkiewicz

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chouikhi, H., Jarray, F. (2023). Sh-DistilBERT: New Transfer Learning Model for Arabic Sentiment Analysis and Aspect Category Detection. In: Nguyen, N.T., et al. Advances in Computational Collective Intelligence. ICCCI 2023. Communications in Computer and Information Science, vol 1864. Springer, Cham. https://doi.org/10.1007/978-3-031-41774-0_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-41774-0_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-41773-3

  • Online ISBN: 978-3-031-41774-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation