Abstract
With the world’s interaction moving more and more toward using online social media platforms, the advent of cyberbullying has also raised its head. Multiple forms of cyberbullying exist from the more common text based to images or even videos, and this paper will explore the context of textual comments. Even in the niche area of considering only text-based data, several approaches have already been worked upon such as n-grams, recurrent units, convolutional neural networks (CNNs), gated recurrent unit (GRU) and even a combination of the mentioned architectures. While all of these produce workable results, the main point of contention is that true contextual understanding is quite a complex concept. These methods fail due to two simple reasons: (i) lack of large datasets to properly utilize these architectures and (ii) the fact that understanding context requires some mechanism of remembering history that is only present in the recurrent units. This paper explores some of the recent approaches to the difficulties of contextual understanding and proposes an ALBERT-based fine-tuned model that achieves state-of-the-art results. ALBERT is a transformer-based architecture and thus even at its untrained form provides better contextual understanding than other recurrent units. This coupled with the fact that ALBERT is pre-trained on a large corpus allowing the flexibility to use a smaller dataset for fine-tuning as the pre-trained model already has deep understanding of the complexities of the human language. ALBERT showcases high scores in multiple benchmarks such as the GLUE and SQuAD showing that high levels of contextual understanding are inherently present and thus fine-tuning for the specific case of cyberbullying allows to use this to our advantage. With this approach, we have achieved an F1 score of 95% which surpasses current approaches such as the CNN + wordVec, CNN + GRU and BERT implementations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Libin, A., Libin, E.: Cyber-anthropology: a new study on human and technological co-evolution. Stud. Health Technol. Inform. 118, 146–156 (2005)
Mishra, S., Diesner, J.: Detecting the correlation between sentiment and user-level as well as text-level meta-data from Benchmark Corpora. In: Proceedings of the 29th on Hypertext and Social Media-HT ’18. pp. 2–10. ACM Press, New York, New York, USA (2018)
Mishra, S., Diesner, J., Byrne, J., Surbeck, E.: Sentiment analysis with incremental human-in-the-loop learning and lexical resource customization. In: Proceedings of the 26th ACM Conference on Hypertext and Social Media-HT ’15. pp. 323–325. ACM Press, New York, New York, USA (2015)
Mishra, S., Agarwal, S., Guo, J., Phelps, K., Picco, J., Diesner, J.: Enthusiasm and support: alternative sentiment classification for social movements on social media. In: Proceedings of the 2014 ACM conference on Web science-WebSci ’14. pp. 261–262. ACM Press, Bloomington, Indiana, USA (2014)
Mishra, S.: Multi-dataset-multi-task neural sequence tagging for information extraction from tweets. In: Proceedings of the 30th ACM Conference on Hypertext and Social Media-HT ’19. pp. 283–284. ACM Press, New York, New York, USA (2019)
Mishra, S., Diesner, J.: Semi-supervised named entity recognition in noisy-text. In: Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT). pp. 203–212. The COLING 2016 Organizing Committee, Osaka, Japan (2016)
Campbell, M.A.: Cyber bullying: an old problem in a new guise. Aust. J. Guidance Couns. 15, 68–76 (2005)
Smith, P.K., Mahdavi, J., Carvalho, M., Fisher, S., Russell, S., Tippett, N.: Cyberbullying: its nature and impact in secondary school pupils. J. Child Psychol. Psychiatry 49, 376–385 (2008)
US legal, available at https://definitions.uslegal.com/c/cyber-bullying/ (2019)
Gaffney, H., Farrington, D.P., Espelage, D.L., Ttofi, M.M.: Are cyberbullying intervention and prevention programs effective? A systematic and meta-analytical review. Aggress. Violent. Beh. 45, 134–153 (2019)
Waseem, Z., Hovy, D.: Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter. Association for Computational Linguistics, pp. 88–93 (2016)
Parent circle, https://www.parentcircle.com/article/cyberbullying-laws-and-policies-in-india/
Nockleby, J.T.: Hate speech. In: Levy, L.W., Karst K.L., et al. (eds) Encyclopedia of the American Constitution, 2nd ed., pp. 1277–1279, Macmillan, New York (2000)
Sood, Sara, Judd Antin, Elizabeth Churchill. Profanity use in online communities. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2012)
Samghabadi, N. S., Maharjan S., Sprague, A., Diaz-Sprague, R., Solorio, T.: Detecting nastiness in social media. In: Proceedings of the First Workshop on Abusive Language Online, pp. 63–72 (2017)
Waseem, Z., Davidson, T., Warmsley, D., Weber, I.: Understanding abuse: a typology of abusive language detection subtasks. In: Proceedings of the First Workshop on Abusive Langauge Online (2017)
Gamb Ack, B., Sikdar, U.K.: Using convolutional neural networks to classify hatespeech. In: Proceedings of the First Workshop on Abusive Language Online, pp. 85–90 (2017)
Gao, L., Huang, R.: Detecting online hate speech using context aware models. arXiv preprint arXiv:1710.07395 (2017)
Zhang, Z., Robinson, D., Tepper, J.: Detecting hate speech on twitter using a convolution-GRU based deep neural network. In: Lecture Notes in Computer Science. Springer, Berlin (2018)
Mozafari, M., Farahbakhsh, R., Crespi, N.: A BERT-based transfer learning approach for hate speech detection in online social media. In: 8th International Conference on Complex Networks and their Applications, Dec 2019, Lisbonne, Portugal, Complex Networks 2019, pp. 928–940 (2019)
Wullach, T., Adler, A., Minkov, E.: Towards hate speech detection at large via deep generative modeling. arXiv preprint arXiv:2005.06370 (2020)
Lan, Z., et al.: Albert: a lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019)
Ross, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., Wojatzki, M.: Measuring the reliability of hate speech annotations: the case of the European Refugee Crisis. In: The 3rd Workshop on Natural Language Processing for Computer-Mediated Communication @ Conference on Natural Language Processing (2016)
Davidson, T., et al.: Automated hate speech detection and the problem of offensive language. In: Eleventh international aaai conference on web and social media (2017)
Hatebase:. Available from: https://hatebase.org/, as on 26 Jul 2020
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems (pp. 5998–6008) (2017)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding (2019)
Liu, Y, Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Tripathy, J.K., Chakkaravarthy, S.S., Satapathy, S.C. et al. ALBERT-based fine-tuning model for cyberbullying analysis. Multimedia Systems 28, 1941–1949 (2022). https://doi.org/10.1007/s00530-020-00690-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00530-020-00690-5