Abstract
The abundance of user-generated online content has presented significant challenges in handling big data. One challenge involves analyzing short posts on social media, ranging from sentiment identification to abusive content detection. Despite recent advancements in pre-trained language models and transfer learning for textual data analysis, the classification performance is hindered by imbalanced data, where anomalous data represents only a small portion of the dataset. To address this, we propose Deep One-Class Fine-Tuning (DOCFT), a versatile method for fine-tuning transfer learning-based textual classifiers. DOCFT uses a one-class SVM-style hyperplane to encapsulate anomalous data. This approach involves a two-step fine-tuning process and utilizes an alternating optimization method based on a custom OC-SVM loss function and quantile regression. Through evaluations on four different hate-speech datasets, we observe that significant performance improvements can be achieved by our method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Antil, H., Brown, T.S., Löhner, R., Togashi, F., Verma, D.: Deep neural nets with fixed bias configuration. arXiv preprint arXiv:2107.01308 (2022)
Basile, V., et al.: Semeval- 2019 task 5: multilingual detection of hate speech against immigrants and women in twitter. In: Proceedings of the 13th International Workshop on Semantic Evaluation, Minneapolis, Minnesota, USA, pp. 54–63. Association for Computational Linguistics (2019)
Bose, S., Su, G.: Deep one-class hate speech detection model. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference. ELRA, Marseille, France, pp. 7040–7048 (2022)
Chalapathy, R., Menon, A., K., Chawla, S.: Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360 (2019)
Chen, J., Hu, Y., Liu, J., Xiao, Y., Jiang, H.: Deep short text classification with knowledge powered attention. In: Proceedings of AAAI 2019/IAAI 2019/EAAI 2019, Honolulu, Hawaii, USA, pp. 6252–6259. AAAI Press (2019)
Dabney, W., Rowland, M., Bellemare, M. G., Munos, R.: Distributional reinforcement learning with quantile regression. In: Proceedings of the AAAI 2018/IAAI 2018/EAAI 2018, New Orleans, Louisiana, USA, pp. 2892–2901. AAAI Press (2018)
Davidson, T., Warmsley, D., Macy, M., Weber, I.: Automated hate speech detection and the problem of offensive language. arXiv preprint arXiv:1703.04009 (2017)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding, In: Proceedings of NAACL-HLT 2019, Minneapolis, Minnesota, pp. 4171–4186. Association for Computational Linguistics (2019)
Duarte, J.M., Berton, L.: A review of semisupervised learning for text classification. Artif. Intell. Rev. (2023). https://doi.org/10.1007/s10462-023-10393-8
Evci, U., Dumoulin, V., Larochelle, H., Mozer, M. C.: Head2toe: utilizing intermediate representations for better transfer learning. arXiv preprint arXiv:2201.03529 (2022)
Fortuna, P., Nunes, S.: A survey on automatic detection of hate speech in text. ACM Comput. Surv. 51(4), 1–30 (2018)
Gibert, O., Perez, N., García-Pablos, A., Cuadros, M.: Hate speech dataset from a white supremacy forum. In: Proceedings of the ALW2, pp. 11–20. Association for Computational Linguistics, Brussels, Belgium (2018)
Hu, C., Feng, Y., Kamigaito, H., Takamura, H., Okumura, M.: One-class text classification with multi-modal deep support vector data description. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3378–3390. Association for Computational Linguistics (2021)
Jahan, M.S., Oussalah, M.: A systematic review of hate speech automatic detection using natural language processing. arXiv preprint arXiv:2106.00742 (2021)
Li, Q., et al.: A survey on text classification: from traditional to deep learning. ACM Trans. Intell. Syst. Technol. 13(2), 1–41 (2022)
Luo, X., Yu, Z., Zhao, Z., Zhao, W., Wang, J.: Effective short text classification via the fusion of hybrid features for IoT social data. Digit. Commun. Netw. 8(6), 942–954 (2022)
Malik, J. S., Pang, G., Hengel, A.: Deep learning for hate speech detection: a comparative study. arXiv preprint arXiv:2202.09517 (2022)
Mandl, T., et al.: Overview of the HASOC track at FIRE 2019: hate speech and offensive content identification in Indo-European languages. In: Proceedings of the FIRE 2019, pp. 14–17. Association for Computing Machinery, New York (2019)
Moya, M.M., Koch, M.W., Hostetler, L.D.: One-class classifier networks for target recognition applications. https://www.osti.gov/biblio/6755553. Accessed 8 Apr 2023
Pang, G., Shen, C., Cao, L., Hengel, A.V.D.: Deep learning for anomaly detection: a review. ACM Comput. Surv. 54(2), 1–38 (2022)
Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana, vol. 1, pp. 2227–2237. Association for Computational Linguistics (2018)
Pitsilis, G.K., Ramampiaro, H., Langseth, H.: Effective hate-speech detection in twitter data using recurrent neural networks. Appl. Intell. 48(12), 4730–4742 (2018)
Poletto, F., Basile, V., Sanguinetti, M., Bosco, C., Patti, V.: Resources and benchmark corpora for hate speech detection: a systematic review. Lang. Resour. Eval. 55(2), 477–523 (2020)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com//openai-assets/research-covers/language-unsupervised/. Accessed 8 Apr 2023
Ruff, L., et al.: Deep one-class classification. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, vol. 80, pp. 4393–4402. PMLR (2018)
Ruff, L., Zemlyanskiy, Y., Vandermeulen, R., Schnake, T., Kloft, M.: Self-attentive, multi-context one-class classification for unsupervised anomaly detection on text. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 4061–4071. ACL (2019)
Schölkopf, B., Alexander, J.S.: Support vector machines, regularization, optimization, and beyond, pp. 656–657. MIT Press (2002)
Sun, M., Dou, H., Yan, J.: Efficient transfer learning via joint adaptation of network architecture and weight. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12358, pp. 463–480. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58601-0_28
Tax, D.M.J.: Data description toolbox. https://homepage.tudelft.nl/n9d04/. Accessed 8 Apr 2023
Tax, D.M.J., Duin, R.P.W.: Support vector data description. Mach. Learn. 54(1), 45–66 (2004)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bose, S., Su, G., Liu, L. (2023). Deep One-Class Fine-Tuning for Imbalanced Short Text Classification in Transfer Learning. In: Yang, X., et al. Advanced Data Mining and Applications. ADMA 2023. Lecture Notes in Computer Science(), vol 14176. Springer, Cham. https://doi.org/10.1007/978-3-031-46661-8_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-46661-8_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46660-1
Online ISBN: 978-3-031-46661-8
eBook Packages: Computer ScienceComputer Science (R0)